Formal Definition of QoE Metrics

This technical report formally defines the QoE metrics which are introduced and discussed in the article “QoE Beyond the MOS: An In-Depth Look at QoE via Better Metrics and their Relation to MOS” by Tobias Hoßfeld, Poul E. Heegaard, Martı́n Varela, Sebastian Möller, accepted for publication in the Springer journal ’Quality and User Experience’ [2]. Matlab scripts for computing the QoE metrics for given data sets are available in GitHub [3]. I. DEFINITION OF QOE METRICS A. Preamble We consider studies where users are asked their opinion on the overall quality (QoE) of a specific service. The subjects (the participants in a study that represent users), rate the quality as a quality rating on a quality rating scale. As a result, we obtain an opinion score by interpreting the results on the rating scale numerically. An example is a discrete 5-point scale with the categories 1,’bad’, 2,’poor’, 3,’fair’, 4,’good’, and 5,’excellent’, referred to as an Absolute Category Rating (ACR) scale [6]. Let U be a random variable (RV) that represents the quality ratings, U ∈ Ω, where Ω is the rating scale, which is also the state space of the random variable U . The RV U can be either discrete, with probability mass function fu, or continuous, with probability density function f(u). The notation used in the paper is summarized in Table I. B. Observations in a study are samples of U In a study, each observation can be regarded as samples Ui,r,c of U . Each sample corresponds to the observation i of subject r, under test condition c, using the rating scale Ω. Typically, the notation can be simplified, e.g., if the identity of a subject is not an issue, and we look at only one test with the same test conditions c, then the set of observations will form a set of samples U = {Ui}, i = 1, · · · , R. More details about test conditions are found in the next section. C. Specification of test conditions A study typically specifies a set of test conditions, C, which includes both technical and non-technical factors. The set of test conditions contains an invariant (static) part that is unchanged over all tests, C0, and a variable (dynamic) part that changes between each of the J tests, Cj , (j = 1, · · · , J). For each test, the test conditions might change according to a pattern specified on a set (of size k), Cj = {cj,1, · · · , cj,k}. If Cj is not the same for all R subjects in test j, then the notation must be extended and indexed with, r, i.e. Cj,r. Furthermore, in some studies we don’t have the same number of subjects per test condition. To specify this, you should add an index to the number of subjects Rj of test j. As an example, consider a study of the effect of a specific sequence of changes in the video quality classes, where each subject is expected to rate the overall technical quality of the video. In each test a specific set of quality classes, qn, is defined, let’s say N classes, in an unordered set Q = {q1, · · · , qN}. The variable part of C is then defined as an ordered set Cj = (cj,1, · · · , cj,k), where c contains both the quality class, q, and the time, t, for the quality class changes, c = (t, q). If the sequence of changes is not the same for each subject in a test, either with respect to q or t, then the notation must capture this by extending and indexing the Cj with r as described above. If the test is about the effect of a number of changes, k, and not a specific sequence of such, then Cj = (k,Q). Finally, if each test only has one condition, Table I KEY VARIABLES AND NOTATIONS USED IN THE PAPER. notation meaning MOS Mean Opinion Score SOS Standard deviation of Opinion Score ACR Absolute Category Rating U random variable (RV) for quality ratings U upper limit of a rating scale U− lower limit of a rating scale U set of ratings {Ui} for a particular test condition R number of subjects (= ratings) per test condition J number of tests S Statistical definition set ({Ω, C,Σ,S}) Ω quality rating scale (same as the sample space of the RV U ) C set of test conditions Σ set of statistics S set of (sufficient) observators u quality rating value for a test condition (e.g. MOS on the ACR scale) fu probability that the quality rating is u ∈ Ω f̂u estimate of fu, i.e. ratio of subjects who rate the test condition with u S(u) SOS as a function of mean opinion score u a SOS parameter of SOS hypothesis in Eq. (11) Aθ α-acceptability, i.e. the probability that ratings are above θ, P (U > θ) %PoW Poor-or-Worse (in %) %GoB Good-or-Better (in %) %TME Terminate Early (in %)